Alphabet Permutation for Differentially Encoding Text

One degree of freedom which is usually not exploited in developing high-performance textprocessing algorithms is the encoding of the underlying atomic character set. Typically, standard character encodings such as ASCII or Unicode are assumed to be a fixed fact of nature, and indeed for most classical string algorithms the assignment of exactly which symbol maps to which k-length bit pattern appears to be an issue of no consequence. In this paper, however, we consider a text compression method where the specific character set collating-sequence employed in encoding the text has a big impact on performance. We demonstrate that permuting the standard character collating-sequences yields a small win on Asian-language texts over gzip. We also show improved compression with our method for English texts, although not by enough to beat standard compression methods. However, we also design a class of artificial languages on which our method clearly beats gzip, often by an order of magnitude. The significance of this work lies partially in evaluating an interesting approach to text compression. Even more, however, we seek to raise awareness of character encodings in the string-algorithms community and ask the question whether alphabet-permutation can lead to improvements in other string and text-processing algorithms.